Search for: All records

Effective options can make reinforcement learning easier by enhancing an agent’s ability to both ex- plore in a targeted manner and plan further into the future. However, learning an appropriate model of an option’s dynamics in hard, requiring estimat- ing a highly parameterized probability distribution. This paper introduces and motivates the Expected- Length Model (ELM) for options, an alternate model for transition dynamics. We prove ELM is a (biased) estimator of the traditional Multi- Time Model (MTM), but provide a non-vacuous bound on their deviation. We further prove that, in stochastic shortest path problems, ELM induces a value function that is sufficiently similar to the one induced by MTM, and is thus capable of support- ing near-optimal behavior. We explore the practical utility of this option model experimentally, finding consistent support for the thesis that ELM is a suit- able replacement for MTM. In some cases, we find ELM leads to more sample efficient learning, espe- cially when options are arranged in a hierarchy.